Papers with computational complexity
KD-Boost: Boosting Real-Time Semantic Matching in E-commerce with Knowledge Distillation (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Existing SOTA techniques for semantic matching are mostly based on Siamese networks. |
| Approach: | They propose a novel knowledge distillation algorithm designed for real-time semantic matching . they train low latency accurate student models by leveraging soft labels from a teacher model . |
| Outcome: | The proposed algorithm outperforms teacher and SOTA knowledge distillation benchmarks on e-commerce datasets. |
Token Level Routing Inference System for Edge Devices (2025.acl-demo)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been gaining in performance but deployment in edge devices faces significant hurdles due to their high computational complexity. |
| Approach: | They propose a collaborative decoding system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. |
| Outcome: | The proposed system achieves 60% performance gain on CommonsenseQA using a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud. |
Attention with Dependency Parsing Augmentation for Fine-Grained Attribution (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing fine-grained attribution methods rely on model-internal similarity metrics but lack a fine-grain representation of the data. |
| Approach: | They propose to use model-internal similarity metrics to validate RAG-generated content . they aggregate token-wise evidence through set union operations and integrate dependency parsing to enrich the semantic completeness of target spans. |
| Outcome: | The proposed method outperforms all prior works in the validation of RAG-generated content. |
Multi-Head Highly Parallelized LSTM Decoder for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | a self-attention network can be easily parallelized at sequence level, but LSTMs are slower to train . a recent study shows that LS models require a lot of computations to perform . |
| Approach: | They propose to compute LSTMs at sequence level to enable sequence-level parallelization . they use a bag-of-words representation of the preceding tokens context to approximate LStms . |
| Outcome: | The proposed model performs better than existing models while being faster to train . the model can be trained efficiently due to the highly parallelized self-attention network . |
SURE: Mutually Visible Objects and Self-generated Candidate Labels For Relation Extraction (2025.coling-main)
Copied to clipboard
| Challenge: | Joint relation extraction models face high computational complexity, complex network architectures, difficult parameter tuning and limited interpretability. |
| Approach: | They develop a candidate label marker mechanism that prioritizes strategic label selection over simple label generation. |
| Outcome: | The proposed candidate label marks improve the SOTA methods by 2.5%, 1.9%, 1.2% . the proposed candidate labels improve the performance of the proposed methods . |
Dual Low-Rank Multimodal Fusion (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing tensor-based fusion methods make poor use of fine-grained temporal dynamics of multimodal sequential features. |
| Approach: | They propose a novel multimodal fusion method called Fine-Grained Temporal Low-Rank Multimodal Fusion that uses low-rank tensor approximation along dual dimensions of input features. |
| Outcome: | The proposed method outperforms the state-of-the-art tensor-based methods with a similar computational complexity. |
Thesis Proposal: Development of End-to-End Speech Translation Models for Indian Languages (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to speech-to-speech translation rely on cascaded pipelines . current approaches rely only on text representations, but they suffer from errors and latency . a new direct speech translation framework is proposed to bridge linguistic gaps . |
| Approach: | They propose a sequence-to-sequence direct speech translation framework that can translate speech from one Indian language to another without relying on intermediate text representations. |
| Outcome: | The proposed framework can translate speech from one Indian language to another without relying on intermediate text representations. |
LOCOST: State-Space Models for Long Document Abstractive Summarization (2024.eacl-long)
Copied to clipboard
Florian Le Bronnec, Song Duong, Mathieu Ravaut, Alexandre Allauzen, Nancy Chen, Vincent Guigue, Alberto Lumbreras, Laure Soulier, Patrick Gallinari
| Challenge: | State-space models are a low-complexity alternative to transformers for text generation . however, the quadratic complexity of the input length restricts the application of large pretrained models to long texts. |
| Approach: | They propose an encoder-decoder architecture based on state-space models for conditional text generation with long context inputs. |
| Outcome: | The proposed model saves memory and memory during training and inference time while saving 50% and 87% of memory. |
LEANCODE: Understanding Models Better for Code Simplification of Pre-trained Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models often require significant computational resources, often constraining input word or code token lengths. |
| Approach: | They propose to use the encoder-decoder attention scores to represent the importance of a code token across multiple contexts to reduce training and prediction time. |
| Outcome: | The proposed approach outperforms the SOTAs DietCode and SlimCode in code search and summarization tasks. |
Long-Short Term Masking Transformer: A Simple but Effective Baseline for Document-level Neural Machine Translation (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning have led to significant improvement of document-level neural machine translation (NMT). |
| Approach: | They propose a long-short term masking self-attention on top of the standard transformer to capture the long-range dependence and reduce the propagation of errors. |
| Outcome: | The proposed model captures the long-range dependence and reduces errors on two publicly available document-level datasets. |
Working Memory Networks: Augmenting Memory Networks with a Relational Reasoning Module (P18-1)
Copied to clipboard
| Challenge: | Recent advances in deep neural networks have enabled complex reasoning tasks. |
| Approach: | They propose a MemNN architecture with a working memory storage and reasoning module that retains relational reasoning abilities of relation networks while reducing computational complexity. |
| Outcome: | The proposed model retains the relational reasoning abilities of the RN while reducing its computational complexity from quadratic to linear. |
Multimodal and Multi-view Models for Emotion Recognition (P19-1)
Copied to clipboard
| Challenge: | combining lexical and acoustic information results in more robust and accurate models . combining both modalities may be a bottleneck in a deployment pipeline due to computational complexity or privacy constraints . |
| Approach: | They propose to combine acoustic and lexical information to provide a deployable acustic model . they use multimodal models and two attention mechanisms to assess the benefits of lexicals . |
| Outcome: | The proposed model outperforms the state-of-the-art on the USC-IEMOCAP dataset . it significantly surpasses models that have been exclusively trained with acoustic features . |
Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies have shown that the effective use of contextual information between sentences can achieve better performance in document-level machine translation. |
| Approach: | They propose a recurrent memory unit to the Transformer to support the information exchange between the sentence and previous context. |
| Outcome: | The proposed model outperforms the previous work on TED and News by 0.91 s-BLEU and 1.49 d-BLUE on average. |
IPED: An Implicit Perspective for Relational Triple Extraction based on Diffusion Model (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to extract relational triples have inherent shortcomings such as redundant information and incomplete triple recognition. |
| Approach: | They propose an Implicit Perspective for relational triple Extraction based on Diffusion model that uses block coverage to complete tables. |
| Outcome: | The proposed method achieves state-of-the-art performance while gaining low computational complexity. |
PCFGs Can Do Better: Inducing Probabilistic Context-Free Grammars with Many Symbols (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent work shows that probabilistic context-free grammars with neural parameterization can be effective in unsupervised constituency parsing. |
| Approach: | They propose a parameterization form of PCFGs based on tensor decomposition which has at most quadratic computational complexity in the symbol number. |
| Outcome: | The proposed model improves unsupervised constituency parsing performance across ten languages. |
A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (2023.emnlp-main)
Copied to clipboard
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
| Challenge: | Existing datasets for webpages contain only fragments of webpages . generative tasks like page description generation and section summarization are often left unstudied . |
| Approach: | They introduce a Wikipedia Webpage suite that contains 2M pages with all associated image, text, and structure data. |
| Outcome: | The proposed approach performs better than full attention with lower computational complexity. |
Tired of Topic Models? Clusters of Pretrained Word Embeddings Make for Fast and Good Topics too! (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing topic models rely on probabilistic models to uncover themes within document collections, but are they the only option? |
| Approach: | They propose a way to cluster pre-trained word embeddings while incorporating document information for weighted clustering and reranking top words. |
| Outcome: | The proposed approach performs as well as classical topic models, but with lower runtime and computational complexity. |
Thinking Long, but Short: Stable Sequential Test-Time Scaling for Large Reasoning Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Inducing models to think for longer can increase accuracy, but as the length of reasoning is further extended, it has also been shown to result in accuracy degradation and model instability. |
| Approach: | They propose a sequential test-time scaling method which induces models to think for longer, but which also generates an increasingly long output. |
| Outcome: | The proposed method improves model accuracy significantly over a wide range of induced thoughts, stabilizing the accuracy of sequential scaling, and eliminating the need for reasoning length fine-tuning. |
Image Embedding Sampling Method for Diverse Captioning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Currently, large-scale captioning models are less accessible for resource-constrained applications such as mobile devices and assistive technologies. |
| Approach: | They propose a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM as the backbone. |
| Outcome: | The proposed framework achieves comparable performance to larger models on MSCOCO, Flickr30k, and Nocaps test datasets while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions. |
Distilling Cross-Modal Knowledge into Domain-Specific Retrievers for Enhanced Industrial Document Understanding (2025.emnlp-industry)
Copied to clipboard
Jinhyeong Lim, Jeongwan Shin, Seeun Lee, Seongdeok Kim, Joungsu Choi, Jongbae Kim, Chun Hwan Jung, Youjin Kang
| Challenge: | Retrieval-Augmented Generation (RAG) has shown strong performance in open-domain tasks, but its effectiveness in industrial domains is limited by a lack of domain understanding and document structural elements (DSE) such as tables, figures, charts, and formula. |
| Approach: | They propose a knowledge distillation framework that transfers complementary knowledge from Large Language Models and Vision-Language Models into a compact domain-specific retriever. |
| Outcome: | The proposed framework outperforms larger baselines while requiring significantly less computational complexity. |
Large Language Models in Bioinformatics: A Survey (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are revolutionizing bioinformatics, enabling advanced analysis of DNA, RNA, proteins, and single-cell data. |
| Approach: | They examine the evolution of Large Language Models (LLMs) in bioinformatics and precision medicine by focusing on genomic sequence modeling, RNA structure prediction, protein function inference, and single-cell transcriptomics. |
| Outcome: | The proposed models are capable of predicting RNA structure and function and predicting single-cell transcriptomics. |
Logits-Based Block Pruning with Affine Transformations for Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods for pruning models rely on calibration data and neglect cumulative effects of pruning on subsequent blocks. |
| Approach: | They propose to use the Logit Disruption Score (LDS) to measure the impact of pruning by comparing the cosine similarity between the logits of the original and pruned models. |
| Outcome: | Experiments on multiple datasets show that the proposed pruning technique reduces reliance on calibration data and improves generalization, achieving competitive results with existing methods. |
Scalable and Accurate Dialogue State Tracking via Hierarchical Sequence Generation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to dialogue state tracking rely on pre-defined ontologies . however, these methods suffer from computational complexity that increases proportionally to the number of pre-determined slots. |
| Approach: | They propose a model that generates a sequence of belief states without the pre-defined ontology list. |
| Outcome: | The proposed model scales easily with the increasing number of pre-defined slots and domains and reaches the state-of-the-art performance on the multi-domain and single domain dialogue state tracking datasets. |
Recurrent Neural Networks as Weighted Language Recognizers (N18-1)
Copied to clipboard
| Challenge: | Recent experiments show that RNNs outperform other methods in assigning high probability to held-out English text. |
| Approach: | They focus on the single-layer, ReLU-activation, rational-weight RNNs with softmax . they show that most problems for such RNN are undecidable . |
| Outcome: | The proposed model outperforms other methods in assigning high probability to held-out English text. |
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors (P18-1)
Copied to clipboard
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency
| Challenge: | Multimodal research is a growing field of artificial intelligence, and fusion is one of the main research problems. |
| Approach: | They propose a low-rank multimodal fusion method which integrates multiple unimodal representations into one compact multimodal representation. |
| Outcome: | The proposed method achieves competitive results on multimodal sentiment analysis, speaker trait analysis, and emotion recognition tasks while reducing computational complexity. |
WIND: Weighting Instances Differentially for Model-Agnostic Domain Adaptation (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for instance weighting cannot learn the weights which make the model generalize well in target domain. |
| Approach: | They propose a modelagnostic instance weighting algorithm which can learn the instance weights instead of manually designed weighting metrics. |
| Outcome: | The proposed method can learn the instance weights instead of manually designed weighting metrics. |
Neural Bi-Lexicalized PCFG Induction (2021.acl-long)
Copied to clipboard
| Challenge: | Neural lexicalized PCFGs make strong independence assumption on the generation of the child word and thus bilexical dependencies are ignored. |
| Approach: | They propose an approach to parameterize L-PCFGs without making implausible independence assumptions. |
| Outcome: | The proposed approach improves both running speed and unsupervised parsing performance on the English WSJ dataset. |
Sharper and Faster mean Better: Towards More Efficient Vision-Language Model for Hour-scale Long Video Understanding (2025.acl-long)
Copied to clipboard
| Challenge: | Existing multimodal large language models (LLMs) have shown impressive performance on the video understanding task, but extremely long videos still pose significant challenges to their context length, memory consumption, and computational complexity. |
| Approach: | They propose a vision-language model named Sophia for long video understanding which can efficiently handle hour-scale long videos. |
| Outcome: | The proposed model exhibits competitive performance compared to existing video understanding baselines across various benchmarks for long video understanding with reduced time and memory consumption. |
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes (2024.acl-long)
Copied to clipboard
| Challenge: | Complex reasoning ability is one of the most important features of Large Language Models. |
| Approach: | They propose a new benchmark that measures the reasoning ability of Large Language Models . it contains 900 algorithmic questions belonging to the NP-Hard complexity class . |
| Outcome: | The proposed benchmark contains 900 questions belonging to the NP-Hard complexity class and is updated on a monthly basis. |
Joint Multi-Decoder Framework with Hierarchical Pointer Network for Frame Semantic Parsing (2021.findings-acl)
Copied to clipboard
| Challenge: | Current researches on frame semantic parsing ignore the interactions among subtasks. |
| Approach: | They propose a multi-decoder strategy to handle these subtasks together . they propose introducing a hierarchical pointer network for argument identification . |
| Outcome: | The proposed architecture improves on state-of-the-art models on FrameNet dataset. |
Finding Influential Instances for Distantly Supervised Relation Extraction (2022.coling-1)
Copied to clipboard
| Challenge: | Distant supervision models suffer from high label noise and are not reliable for DS. |
| Approach: | They propose a model-agnostic instance sampling method for relation extraction (RE) by influence function, namely REIF. |
| Outcome: | The proposed method reduces the computational complexity from O(mn) to O(1), with analyzing its robustness on the selected sampling function. |
Fast End-to-end Coreference Resolution for Korean (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent advances in coreference resolution have come at a cost of computational complexity and have not been addressed. |
| Approach: | They propose a pointer network that leverages the linguistic property of head-final languages to reduce coreference linking search space and achieve 2x speedup in document processing time. |
| Outcome: | The proposed model maintains state-of-the-art performance 66.9% of CoNLL F1 on ETRI test set while achieving 2x speedup (30 doc/sec) in document processing time. |
Identifying the limits of transformers when performing model-checking with natural language (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies have focused on transformer models’ ability to perform reasoning on text, but the above question has not been adequately answered. |
| Approach: | They investigated the problem of model-checking with natural language to determine whether transformers can comprehend logical semantics in natural language. |
| Outcome: | The proposed model-checking problem is suited to address this issue but is untouched in natural language inference research. |
The NLP Task Effectiveness of Long-Range Transformers (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing benchmarks on long-range attention models have not been sufficient to develop efficient Transformers and their practical application on complex NLP tasks. |
| Approach: | They propose to benchmark 7 Transformer variants on 5 difficult NLP tasks and 7 datasets to examine their capacity for long-range attention. |
| Outcome: | The proposed models have advantages on content selection and query-guided decoding, but they come with previously unrecognized drawbacks such as insufficient attention to distant tokens and accumulated approximation error. |
Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to reduce dataset bias rely on spurious correlations and obstruct valid feature information while mitigating bias. |
| Approach: | They propose a representation normalization method which disentangles correlations between features of encoded sentences and a kernel approximation method which provides isotropic data distribution. |
| Outcome: | The proposed method eliminates the bias problem by providing isotropic data distribution while maintaining in-distribution accuracy. |
HadSkip: Homotopic and Adaptive Layer Skipping of Pre-trained Language Models for Efficient Inference (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to exit pre-trained language models suffer from the limitation that they have to sequentially traverse through all layers prior to the selected exit layer, which degrades their performance. |
| Approach: | They propose a homotopic and adaptive layer skipping fine-tuning method that adaptively selects the layers to skip based on a predefined budget. |
| Outcome: | The proposed method outperforms all state-of-the-art baselines on the GLUE benchmark and shows that it is highly efficient. |
Rethinking the Multimodal Correlation of Multimodal Sequential Learning via Generalizable Attentional Results Alignment (2024.acl-long)
Copied to clipboard
| Challenge: | Existing studies have focused on the alignment of multimodal sequential learning using transformers. |
| Approach: | They propose a constrained scheme to align the multiple attentional results from both local and global perspectives. |
| Outcome: | The proposed scheme could align the multiple attentional results from both local and global perspectives, making the information capture more efficient. |
Ranking-Based Autoencoder for Extreme Multi-label Classification (N19-1)
Copied to clipboard
| Challenge: | Existing methods to solve label dependency and noisy labeling problems are limited . experimental results show the proposed method is competitive to state-of-the-art methods . |
| Approach: | They propose a deep learning XML method with word-vector-based self-attention followed by ranking-based AutoEncoder architecture to solve these problems. |
| Outcome: | The proposed method is competitive to state-of-the-art methods on benchmark datasets. |
Accelerated High-Quality Mutual-Information Based Word Clustering (2020.lrec-1)
Copied to clipboard
| Challenge: | Word clustering is a hard hierarchical clustering that uses short-range distributional information to construct clusters. |
| Approach: | They propose to use a hierarchical clustering algorithm with a fixed-width beam to build clusters that outperform other word representations. |
| Outcome: | The proposed method outperforms the original methods in the computation of hierarchical and flat clusters. |
Chain-of-Rank: Enhancing Large Language Models for Domain-Specific RAG in Edge Device (2025.findings-naacl)
Copied to clipboard
| Challenge: | Retrieval-augmented generation (RAG) is valuable in specialized domains where precision is critical. |
| Approach: | They propose a chain-of-rank algorithm which allows LLMs to access a target domain early via finetuning. |
| Outcome: | The proposed method achieves state-of-the-art in benchmarks and analyzes its efficacy. |
Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammars (2023.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that context-free grammars are not natural for modeling discontinuous language phenomena such as extrapositions and cross-serial dependencies. |
| Approach: | They propose a grammar induction approach with mildly context-sensitive grammars for unsupervised discontinuous parsing. |
| Outcome: | Experiments on German and Dutch show that the proposed grammar induction method is beneficial for unsupervised parsing. |
SparseFlow: Accelerating Transformers by Sparsifying Information Flows (2024.acl-long)
Copied to clipboard
| Challenge: | SparseFlow is an efficient method to sparsify the dense information flows within transformers. |
| Approach: | They propose a method to sparsify the dense pathways of token representations across all transformer blocks by parameterizing them to be sparse. |
| Outcome: | The proposed method reduces computational costs by half on average without compromising task accuracy. |
Scalable Syntax-Aware Language Models Using Knowledge Distillation (P19-1)
Copied to clipboard
| Challenge: | Prior work has shown that syntactic neural language models learn from small amounts of training data more effectively than sequential models. |
| Approach: | They propose a knowledge distillation technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model and enables it to develop a more structurally sensitive representation of the larger training data. |
| Outcome: | The proposed method improves on baseline syntactic evaluations on LSTMs with a higher level of accuracy than previous methods. |
Flooding-X: Improving BERT’s Resistance to Adversarial Attacks via Loss-Restricted Fine-Tuning (2022.acl-long)
Copied to clipboard
Qin Liu, Rui Zheng, Bao Rong, Jingyi Liu, ZhiHua Liu, Zhanzhan Cheng, Liang Qiao, Tao Gui, Qi Zhang, Xuanjing Huang
| Challenge: | Existing approaches to generating adversarial perturbations scale up the cost of training computational complexity by the number of gradient steps it takes to obtain the adversarials. |
| Approach: | They propose a flood method which aims at better generalization and a criterion to bring hyper-parameter-dependent flooding into effect with a narrowed-down search space by measuring how the gradient steps taken within one epoch affect the loss of each batch. |
| Outcome: | The proposed method improves BERT’s resistance to textual adversarial attacks by a large margin and achieves state-of-the-art robust accuracy on various text classification and GLUE tasks. |
On the Transformer Growth for Progressive BERT Training (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing methods only conduct network growth in a single dimension, but compound growth operators are beneficial for multiple dimensions. |
| Approach: | They propose a method to train BERT progressively using a Transformer model and explore alternative growth operators in each dimension via controlled comparison. |
| Outcome: | The proposed method speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively while achieving comparable performances. |
GLiREL - Generalist Model for Zero-Shot Relation Extraction (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing approaches to zero-shot named entity recognition rely on distant supervision and training data for unseen labels. |
| Approach: | They propose an efficient architecture and training paradigm for zero-shot relation classification . they use a protocol to generate multiple relation labels in a single forward pass . |
| Outcome: | The proposed architecture and training paradigm achieve state-of-the-art results on the zero-shot relation classification task. |
EBERT: Efficient BERT Inference with Dynamic Structured Pruning (2021.findings-acl)
Copied to clipboard
| Challenge: | Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. |
| Approach: | They propose a dynamic structured pruning algorithm that prunes model weights at run-time . they propose to prune the unimportant heads in multi-head self-attention layers . |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods on different tasks. |
SalaMAnder: Shapley-based Mathematical Expression Attribution and Metric for Chain-of-Thought Reasoning (2025.findings-emnlp)
Copied to clipboard
Yue Xin, Chen Shen, Shaotian Yan, Xiaosong Yuan, Yaoming Wang, Xiaofeng Zhang, Chenxi Huang, Jieping Ye
| Challenge: | Chain-of-Thought prompting improves the math reasoning capability of large language models. |
| Approach: | They propose a method for attribution of component-level contributions in CoT reasoning using Shapley value and a stratified sampling algorithm that significantly reduces computational complexity. |
| Outcome: | The proposed method reduces computational complexity and provides robust correlations with model performance. |
GLoCIM: Global-view Long Chain Interest Modeling for news recommendation (2025.coling-main)
Copied to clipboard
| Challenge: | Recent efforts to extract local subgraph information from click graphs have hindered collaboratively utilizing global click graph information. |
| Approach: | They propose a global-view long chain interests model that models a click graph with neighbor interest to enhance news recommendation. |
| Outcome: | The proposed method surpasses baseline methods on two real-world datasets. |
AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference Models (2025.findings-acl)
Copied to clipboard
Qi Liu, Jingqing Ruan, Hao Li, Haodong Zhao, Desheng Wang, Jiansong Chen, Wan Guanglu, Xunliang Cai, Zhi Zheng, Tong Xu
| Challenge: | Existing multi-objective preference alignment methods for large language models face limitations such as auxiliary reward/reference models and computational complexity. |
| Approach: | They propose a framework that achieves dynamic balance across preference dimensions by using dimension-aware generation metrics as implicit rewards. |
| Outcome: | Empirical results show that AMoPO outperforms state-of-the-art methods by 28.5% . |
DenseSSM: State Space Models with Dense Hidden Connection for Efficient Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) face excessive computational and memory requirements due to the commonly used Transformer architecture. |
| Approach: | They propose a method to enhance the flow of hidden information between layers in large language models by selectively integrating shallow-layer hidden states into deeper layers. |
| Outcome: | The proposed method maintains parallelizability and inference efficiency of SSMs while significantly boosting performance on public benchmarks. |
DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-off (2025.emnlp-main)
Copied to clipboard
Jusheng Zhang, Yijia Fan, Kaitong Cai, Zimeng Huang, Xiaofei Sun, Jian Wang, Chengpei Tang, Keze Wang
| Challenge: | et al., 2019; Brown e.t al, 2023; Touvron e t al; 2024; OpenAI, 2024) Large Language Models (LLMs) have demonstrated remarkable capabilities in knowledge encoding and contextual understanding during their pretraining phase. |
| Approach: | They propose a dynamic expert scheduling mechanism that allocates computational resources based on text complexity and a hierarchical sparse attention mechanism that adjusts attention patterns according to a variety of input lengths. |
| Outcome: | The proposed framework overpowers existing methods on long-text generation benchmarks. |
Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models (D18-1)
Copied to clipboard
| Challenge: | In this paper, we propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) in neural languages modelling, called dual-FOFE. |
| Approach: | They propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) in neural languages modelling, called dual-FOFE. |
| Outcome: | The proposed method significantly reduces the complexity and improves perplexity by 10% over the original FOFE model. |
Learning to Infer Entities, Properties and their Relations from Clinical Conversations (D19-1)
Copied to clipboard
| Challenge: | Existing relation extraction models restrict inferring relations between tokens within a few neighboring sentences to avoid high computational complexity. |
| Approach: | They propose a Span Attribute Tagging (SAT) model to infer clinical entities and their properties using a hierarchical two-stage approach. |
| Outcome: | The proposed model outperforms baseline models in identifying relations between symptoms and properties by about 32% and 50% on medications and their properties. |
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020.acl-main)
Copied to clipboard
| Challenge: | Recent research shows that transformer-based neural networks can greatly advance the state of the art over many natural language processing tasks. |
| Approach: | They propose a technique to adapt transformer-based models into a cascade of rankers. |
| Outcome: | The proposed technique reduces computation by 37% with almost no impact on accuracy on two English question answering datasets. |
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts (2025.naacl-long)
Copied to clipboard
Zhenpeng Su, Xing W, Zijia Lin, Yizhe Xiong, Minxuan Lv, Guangyuan Ma, Hui Chen, Songlin Hu, Guiguang Ding
| Challenge: | Large language models (LLMs) have been attracting much attention due to their impressive performance in all kinds of downstream tasks. |
| Approach: | They propose a mix-of-experts model that allows the model size to grow without raising training costs. |
| Outcome: | The proposed model outperforms existing models in perplexity and robustness tests. |
LoRMA: Low-Rank Multiplicative Adaptation for LLMs (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models have shown impressive generalization capabilities, but can be expensive to fine-tune due to high computational costs. |
| Approach: | They propose a low-rank multiplicative Adaptation technique that shifts the paradigm of additive updates to a richer space of matrix multiplicative transformations. |
| Outcome: | The proposed approach overcomes computational complexity and rank bottlenecks in terms of matrix multiplication metrics. |
Adaptive Attention for Sparse-based Long-sequence Transformer (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent studies show that Transformers can process longer sequences because of their complexity and time scales quadratic to the sequence length. |
| Approach: | They propose an efficient Transformer model with adaptive attention that can select useful tokens automatically in sparse attention by learnable position vectors. |
| Outcome: | The proposed model can select useful tokens automatically in sparse attention by learnable position vectors. |
Segmented Recurrent Transformer: An Efficient Sequence-to-Sequence Model (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Transformers have shown dominant performance across a range of domains including language and vision, but their computational cost grows quadratically with the sequence length, making their usage prohibitive for resource-constrained applications. |
| Approach: | They propose a segmented recurrent transformer that combines segmente recursion with recursive attention to reduce the computational cost. |
| Outcome: | The proposed model achieves higher ROUGE1 scores and lower computational complexity than current approaches. |
Evaluating Word Expansion for Multilingual Sentiment Analysis of Parliamentary Speech (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent efforts to create and format data sets of parliamentary speech material have facilitated cross-lingual comparisons and highlighted the need for methods that are computationally efficient and language-agnostic. |
| Approach: | They propose a word expansion method for sentiment lexicon generation that leverages word embeddings and vector similarity to expand synonym seed lists with domain-specific terms from the speech corpora. |
| Outcome: | The proposed method is compared with other multilingual lexica and is highly sensitive to processing and scoring techniques. |
ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing MCIT methods do not fully exploit the unique attribute of Large Multimodal Models and often gain performance at the expense of efficiency. |
| Approach: | They propose a multimodal continual instruction learning framework that exploits the ability of LMMs to learn mixed instruction datasets and prompts for each task. |
| Outcome: | The proposed framework achieves +14.26% performance gain on MCIT benchmarks with remarkable x1.42 inference speed free from growing computation. |
Parallel Refinements for Lexically Constrained Text Generation with BART (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing work injects lexical constraints into the output, which generates generic or ungrammatical sentences and has high computational complexity. |
| Approach: | They propose a model that incorporates pre-specified keywords into the output to control the generated text. |
| Outcome: | The proposed model decomposes the generated text into two sub-tasks and improves the sentence quality. |
PECAN: LLM-Guided Dynamic Progress Control with Attention-Guided Hierarchical Weighted Graph for Long-Document QA (2025.findings-acl)
Copied to clipboard
| Challenge: | Long-document Question Answering (QA) challenges with large-scale text and long-distance dependencies. |
| Approach: | They propose a method that leverages large language models to control retrieval process . they propose 'attention-based' retrieval methods that construct hierarchical graphs . |
| Outcome: | The proposed method achieves LLM-level performance while maintaining computational complexity comparable to RAG methods. |
How Do Hyenas Deal with Human Speech? Speech Recognition and Translation with ConfHyena (2024.lrec-main)
Copied to clipboard
| Challenge: | Currently, attention-based models face computational hurdles in processing long sequences due to its quadratic complexity. |
| Approach: | They propose a conformer whose encoder self-attentions are replaced with Hyena for speech processing . they propose 'confhyena' model that reduces training time by 27% at minimal cost . |
| Outcome: | The proposed model reduces training time by 27% at the cost of minimal quality degradation. |
Connectivity Patterns are Task Embeddings (2023.findings-acl)
Copied to clipboard
Zhiheng Xi, Rui Zheng, Yuansen Zhang, Xuanjing Huang, Zhongyu Wei, Minlong Peng, Mingming Sun, Qi Zhang, Tao Gui
| Challenge: | Existing methods for predicting inter-task transferability are sparse and task-specific. |
| Approach: | They propose a method that uses connectivity patterns of neurons as a unique identifier associated with a task. |
| Outcome: | The proposed method outperforms baselines in predicting inter-task transferability across data regimes and transfer settings while keeping high efficiency in computation and storage. |
RWKV: Reinventing RNNs for the Transformer Era (2023.findings-emnlp)
Copied to clipboard
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Gv, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartłomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan Wind, Stanisław Woźniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, Rui-Jie Zhu
| Challenge: | recurrent neural networks struggle to match the performance of Transformers due to limitations in parallelization and scalability. |
| Approach: | They propose a model architecture that combines the efficient parallelizable training of transformers with the efficient inference of RNNs. |
| Outcome: | The proposed model performs on par with similarly sized RNNs, suggesting future work can leverage this architecture to create more efficient models. |
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Considerable efforts have been and are still being put into increasing the context length of Large Language Models (LLMs) |
| Approach: | They propose an approach that divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. |
| Outcome: | The proposed approach outperforms 8 state-of-the-art methods in effectiveness and efficiency for document summarization and question answering, and achieves the best performance on LongBench v2 among models of comparable size. |
MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection (2025.acl-long)
Copied to clipboard
| Challenge: | MaCP is a new adaptation method for large foundation models that requires minimal parameters and memory for fine-tuning. |
| Approach: | They propose a method that exploits the superior energy compaction and decorrelation properties of cosine projection to improve model efficiency and accuracy. |
| Outcome: | The proposed method improves model efficiency and accuracy across a wide range of single-modality tasks including natural language understanding, natural language generation, text summarization, and multi-modalities such as image classification and video understanding. |
Attention-guided Self-reflection for Zero-shot Hallucination Detection in Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Hallucination is a significant barrier to the effective application of Large Language Models (LLMs). |
| Approach: | They propose an Attention-Guided SElf-Reflection approach for hallucination detection in Large Language Models. |
| Outcome: | The proposed method significantly outperforms existing methods in zero-shot hallucination detection on four widely-used LLMs across three different halluciation benchmarks. |
EasyEA: Large Language Model is All You Need in Entity Alignment Between Knowledge Graphs (2025.findings-acl)
Copied to clipboard
| Challenge: | Entity alignment (EA) aims to identify entities in different knowledge graphs (KGs) that represent the same real-world object. |
| Approach: | They propose an end-to-end EA framework based on large language models that requires no training to implement. |
| Outcome: | The proposed framework significantly reduces the reliance on seed entity pairs while achieving state-of-the-art (SOTA) performance on diverse datasets. |
SpecHub: Provable Acceleration to Multi-Draft Speculative Decoding (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have limited inference speed due to sequential token generation . Spechub is a novel, efficient sampling-verification method for MDSD that improves acceptance rates with only linear computational overhead. |
| Approach: | They propose a method that uses a smaller draft model to generate multiple token sequences . Spechub generates 0.05-0.27 and 0.02-0.16 more tokens per step than RRS and RRS without replacement . |
| Outcome: | The proposed method improves acceptance rates with only linear computational overhead. |
GRaMPa: Subword Regularisation by Skewing Uniform Segmentation Distributions with an Efficient Path-counting Markov Model (2025.acl-long)
Copied to clipboard
| Challenge: | Subword regularisations are known to be stochastic, but only a handful of possible segmentations are sampled. |
| Approach: | They propose to randomise word segmentations from a subword tokeniser instead of randomising them by weighting paths in an unweighted segmentation graph. |
| Outcome: | The proposed method outperforms existing methods on token-level tasks with spelling errors. |
Language Fusion for Parameter-Efficient Cross-lingual Transfer (2025.acl-long)
Copied to clipboard
| Challenge: | Limited availability of multilingual text corpora for pretraining results in poor performance on downstream tasks due to undertrained representation spaces for languages other than English. |
| Approach: | They propose a method that integrates source and target language representations within low-rank (LoRA) adapters using lightweight linear transformations to enhance representation quality and transfer performance for languages other than English. |
| Outcome: | The proposed method improves representation quality and performance for languages other than English while maintaining parameter efficiency. |
Semantic Map-based Generation of Navigation Instructions (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing approaches to navigation instruction generation use a sequence of panorama images as visual input. |
| Approach: | They propose a new approach to navigation instruction generation using semantic maps as visual input and frame it as an image captioning task. |
| Outcome: | The proposed model is based on a dataset of a human vision and language navigation task and human subjects are asked to manually assess the quality of the generated instructions. |
SMEC:Rethinking Matryoshka Representation Learning for Retrieval Embedding Compression (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large language models generate high-dimensional embeddings that capture rich semantic and syntactic information. |
| Approach: | They propose a training framework to reduce dimensionality and complexity of large language models. |
| Outcome: | Experiments on image, text, and multimodal datasets show that the proposed training framework reduces dimensionality while maintaining performance. |
LLM-based Rumor Detection via Influence Guided Sample Selection and Game-based Perspective Analysis (2025.acl-long)
Copied to clipboard
Zhiliang Tian, Jingyuan Huang, Zejiang He, Zhen Huang, Menglong Lu, Linbo Qiao, Songzhu Mei, Yijie Wang, Dongsheng Li
| Challenge: | Existing methods for rumor detection on social media are limited by limited modeling capacity and insufficient training corpora. |
| Approach: | They propose an SFT-based rumor detection model with Influence guided Sample selection and Game-based multi-perspective analysis to address these issues. |
| Outcome: | The proposed model outperforms existing SOTA on three datasets. |
Less is More: Knowledge-Aware Compression for Long Legal Judgment Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances leverage large language models (LLMs) for legal reasoning, but they face high computational costs and information degradation when handling long cases. |
| Approach: | They propose a framework that selectively retains legally relevant information while reducing redundant or less informative content, enabling efficient and accurate long-context reasoning. |
| Outcome: | The proposed framework outperforms existing methods on four real-world datasets spanning multiple jurisdictions and languages. |
Two-Stage Parameter Alignment for Multi-LoRA Merging in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Current general model merging methods are prone to parameter interference problems . a novel two-stage parameter alignment framework is proposed to address this problem . |
| Approach: | They propose a two-stage parameter alignment framework that integrates low-rank LoRAs . they propose to reduce the computational complexity of existing methods by preserving fine-grained functions . |
| Outcome: | The proposed framework exhibits greater robustness than other methods in high-rank and high-interference scenarios while preserving fine-grained functions. |
Cross-MoE: An Efficient Temporal Prediction Framework Integrating Textual Modality (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models ignore dynamic and different relations between time series patterns and textual features, which leads to poor performance in temporal-textual feature fusion. |
| Approach: | They propose a temporal-textual fusion framework that replaces Cross Attention with Cross-Ranker to reduce computational complexity and enhances modality-aware correlation memorization with Mixture-of-Experts (MoE) networks to tolerate the distributional shifts in time series. |
| Outcome: | The proposed framework reduces MSE by 8.78% compared to the current SOTA model and requires only 75% of computational overhead and 12.5% of activated parameters. |
ConvX: A Lightweight Converter to Bridge Indexed Dense Representations and Large Language Models for Retrieval-Augmented Generation (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing RAG pipelines suffer from critical efficiency limitations due to their complexity and complexity. |
| Approach: | They propose a compression-based RAG framework that directly leverages indexed dense representations produced by a retriever, substituting to long text contexts. |
| Outcome: | Empirical results show that the proposed model achieves competitive performances compared to the state-of-the-art model that uses a large ad-hoc context compressor while offering substantially improved inference efficiency. |